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Abstract 

Time  needed  to  develop  and  field  new  military  capabilities  is  becoming  an  increasingly 
serious  problem.  Among  other  things,  development  times  have  steadily  increased.  In  this 
paper,  we  attempt  to  structure  the  schedule  estimating  problem,  present  some  initial  results, 
and  propose  a  research  agenda  to  improve  schedule  estimating.  Accordingly,  we  seek 
preliminary  answers  to  the  following  questions. 

•  What  is  the  current  state  of  the  art  for  estimating  acquisition  program  schedules? 
What  should  it  be? 

•  What  are  salient  features  of  program  management  trade-offs,  especially  between 
schedule  and  cost  (which  are  related  in  complex,  imperfectly  understood  ways)? 
In  what  areas  should  air  combat  performance  measures  need  updating? 

•  What  are  the  elements  of  a  research  agenda  for  learning  more  about  schedule 
estimating? 

We  also  present  some  preliminary  results  in  the  form  a  narrative  case  study  of  the  F-35 
program  and  empirical  estimates  of  schedules. 

The  JCIDS  (Joint  Capability  Integration  and  Development  System)  Instruction  recommends 
“effective  cost,  performance,  schedule  and  quantity  trade-offs”  as  being  highly  conducive  to 
successful  acquisition  programs  (CJCS,  2015,  p.  A-9).  However,  these  attributes  have 
received  rather  unequal  interest — with  cost  garnering  the  most  attention. 

For  example,  in  the  DoD’s  latest  acquisition  performance  report  (DoD,  2015),  “cost”  appears 
1 8  times  in  the  table  of  contents  and  86  times  in  the  highlights;  “schedule”  appears  six  times 
in  table  of  contents  and  37  times  in  the  highlights.  (“Operational  performance”  appears  only 
six  times  in  the  contents.)  In  our  conference  program,  “cost”  appears  14  times  in  five 
sessions,  while  “schedule”  appears  four  times,  in  one  session. 
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Cost  is  certainly  important,  and  warrants  much  attention,  which  it’s  received.  The  DoD  has 
devoted  considerable  time,  attention  and  resources  to  more  realistic  cost  estimating.  And, 
seems  to  us,  there’s  been  a  great  deal  of  progress  toward  that  goal. 

But  schedule  is  also  important  and  is  becoming  more  so.  With  multiple  Revolutions  in  Military 
Affairs  ongoing  simultaneously,  we  have  entered  a  hyper-adaptive  era  in  military  affairs — 
enabled,  inter  alia,  by  rapid  advances  in  information  technology.  As  Deputy  Secretary  Work 
has  stated,  innovators  now  encounter  “fast  followers”  (Freedburg,  2015).  Accordingly,  the 
operational  implications  of  longer  schedules  have  indeed  become  more  important.  And  the 
DoD’s  leadership  recognizes  that  importance.  A  number  of  organizations  have  recently  been 
created  in  order  to  field  new  capabilities  more  quickly. 

In  short,  major  changes  in  international  military  affairs  and  recent  DoD  emphasis  has  created 
a  new  environment,  in  which  an  ability  to  understand  the  schedule  consequences  of  program 
strategies  is  especially  important. 

Accordingly,  we  discuss  the  matter  of  acquisition  schedules  within  the  context  of 
contemporary  military  affairs  below.  Then  we  address  schedule  estimating  tools  and 
schedule  estimating  methods.  In  the  section  following  that,  we  take  the  JCIDS  instruction 
literally,  and  essay  an  abstract  discussion  of  cost-time-performance  trade-offs. 

One  promising  variable  for  schedule  estimating  relationships  is  system  performance,  which  is 
discussed  next — primarily  in  the  context  of  tactical  fighters,  the  F-35  in  particular. 

The  section  titled  Toward  Explaining  the  Time  Curve1  is  about  empirical  models  for  estimating 
schedules.  One  likely  schedule  driver  is  requirements  growth.  In  a  following  section,  we  offer 
a  narrative  concerning  the  requirements  growth  that  occurred  from  CALF  (Common 
Affordable  Lightweight  Fighter)  to  JSF  (Joint  Strike  Fighter,  F-35).  Finally,  we  offer  concluding 
comments  and  thoughts  about  a  research  agenda  aimed  at  making  more  realistic  schedule 
estimating  tools  available  to  our  acquisition  professionals. 

Introduction 

Why  Schedules  Are  Important 

“The  fact  is  that  we  are  slower  than  the  bad  guys.” 

— Esti  Peshin,  Director  of  Cyber  Programs  for  Israel  Aerospace  Industries 
(quoted  in  Sternstein,  2015). 

Cost  and  schedule  are  critical  variables  in  any  acquisition  program.  And  the  DoD  has 
indeed  committed  serious  efforts  over  an  extended  period  of  time  to  develop  the  means  for 
realistic  acquisition  cost  estimates. 

There  are  at  least  five  good  reasons  for  increased  focus  on  realistic  schedule 
estimates: 

•  Planning  to  pay  for  force  modernization  in  an  era  of  restrained  budgets, 
especially  in  the  next  decade; 

•  Longer  times  to  field  new  capabilities  (absolute  and  relative); 

•  Rapid  fielding  initiatives  throughout  the  DoD;  and 


1  That  is,  the  upward  trend  in  time  needed  to  field  new  systems. 
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•  Current  marching  orders,  including  the  Air  Force  “should-schedule”  initiative. 

(Apparently)  Looming  Budget  Squeeze 

There’s  a  budget  squeeze  for  the  DoD  expected  in  the  2020s,  driven  largely  by 
modernization  programs.  This  has  been  well  documented  by  experts  both  inside  and  outside 
the  government  (e.g.,  Congressional  Research  Service  [Gertler,  2015];  Center  for  Strategic 
and  International  Studies  [Harrison,  2016]). 

This  is  especially  difficult  for  the  Air  Force,  whose  top-3  priority  acquisition  programs 
account  for  almost  all  of  resources  expected  to  be  available  for  modernization.  For  example, 
these  (KC-46,  F-35,  Long-Range  Strike  Bomber),  plus  C-130J  and  unmanned  aircraft 
account  for  99%  of  the  service’s  aircraft  acquisition  budget  for  FY16  (Gertler,  2015, 
Summary,  1),  with  the  situation  continuing  throughout  the  2020s. 

Furthermore,  there’s  every  reason  to  expect  budget  squeezes  to  continue  well  into 
the  future.  The  entitlement  bills  are  now  coming  due — expected  to  account  for  about  15%  of 
GDP  in  2026.  Net  interest  on  the  federal  debt  is  estimated  at  3%;  “discretionary” 
expenditures  for  about  5%,  about  half  of  which  is  estimated  for  defense  (Congressional 
Budget  Office  [CBO],  2016,  esp.  pp.  66,  84).  Revenues  are  estimated  at  about  18%  of  GDP 
(CBO,  2016,  p.  92),  with  long-term  deficits  of  about  5%  of  GDP.  This  means  major  pressure 
on  the  “discretionary”  categories — defense  especially. 

Therefore,  some  preplanning  and  painful  prioritizing  seems  both  necessary  and 
inevitable.  A  number  of  options  are  available,  none  of  them  pleasant  (Gertler,  2015;  Hale, 
2016;  Harrison,  2016).  And  it’s  reasonable  to  expect  that  the  longer  necessary  decisions  are 
postponed,  the  range  of  alternatives  available  will  continue  to  narrow.  But  without 
reasonably  good  program  schedule  estimates,  any  early  decision  loses  credibility  and 
usefulness. 

Schedules  Have  Become  More  Important:  Time  to  Deliver  New  Systems  Is  Increasing 

“(Acquisition)  lead  time  in  the  U.S.  is  too  long,”  according  to  LTG  Arthur  Trudeau, 
Army  Chief  of  Research  and  Development  (1958,  quoted  in  Peck  &  Scherer,  1962,  p.  425). 
But  lead  times  are  getting  longer.  For  example,  the  F-35  concept  is  generally  regarded  as 
being  formed  in  July  1993,  with  the  creation  of  the  Joint  Advanced  Strike  Technology  (JAST) 
program  (Defense  Science  Board  [DSB],  1994,  ES-1).  The  F-35B,  for  example,  was 
declared  operational  on  July  31,  2015  (USMC,  2015),  meaning  a  lead  time  of  22  years. 

This  is  illustrated  in  Figure  1.  One  implication  is  that  the  widely-mentioned  2030  IOC2 
of  a  next-generation  fighter  aircraft  appears  fanciful  at  best.  If  source  selection  for  a  sixth- 
generation  fighter  aircraft  occurs  in  2020  (optimistic),  we  can  expect  an  IOC  some  time  past 
the  middle  of  the  2030s. 

Schedules  Are  Becoming  More  Important  in  an  Era  of  Faster  Followers 

As  Robert  Work,  Deputy  Secretary  of  Defense,  put  it,  we’re  in  an  era  of  “fast 
followers”  in  military  affairs  (Freedburg,  2015).  And  there’s  excellent  reason  this  problem  is 


2  Fielding  a  new  fighter  in  2030  has  been  advocated  as  an  operational  “requirement”  (Gen  Mike 
Hostage,  quoted  in  Mehta,  2012)  and  also  to  alleviate  fighter  aircraft  shortfalls  (Tirpak,  2009,  38). 
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getting  worse;  as  an  Air  Force  flag  officer  put  it,  “Emerging  threats’  timelines  are  decreasing. 
(Our)  acquisition  times  are  increasing.”3 

The  current  schedule  difficulty  is  made  more  acute  due  to  the  adaptiveness  of  our 
rivals.  Given  especially  the  extended  period  of  development  for  many  U.S.  weapon  systems, 
those  countermeasures  have  time  to  development.  Thus,  for  example,  potential  adversaries 
have  seriously  pursued  countermeasures  to  U.S.  stealth  fighters  (e.g.,  Fulghum,  2012; 
Keller,  2016;  Majumdar,  2014;  Sweetman,  2015c,  2015d).  The  principal  enabling  technology 
is  rapid  computing,  which  can  combine  fragmentary  sensor  information  into  a  unified  picture 
(Clark,  2014).  All  in  all,  stealth  may  indeed  be  overrated  (as  the  CNO,  Admiral  Jonathan 
Greenert,  stated,  quoted  in  Hasik,  2016a). 


Figure  1.  The  Time  Curve  in  Months  to  IOC  vs.  Source  Selection  Year  for  Tactical 

Fighters 

(expanded  from  Blickstein  et  al. ,  201 1 ,  Table  4.5,  p.  48) 

This  is  relevant  to  schedules.  It’s  possible  that  a  stealthy  aircraft,  if  delayed  long 
enough,  can  operate  only  at  considerably  reduced  effectiveness  (Franck  et  al.,  2012,  p.  68). 
Therefore,  it’s  important  that  new  capabilities,  and  upgrades,  be  fielded  in  a  timely  manner 
and  that  planners  have  a  realistic  estimate  of  how  long  it  will  take  to  field  new  combat 
capabilities. 

Current  Marching  Orders  Regarding  Schedules 

Recognizing  the  problems  discussed  above,  the  services  and  the  DoD  have 
undertaken  initiatives  to  field  new  capabilities  sooner.  These  have  included  the  Air  Force 
Office  of  Rapid  Capabilities  (RCO)  and  OSD’s  Strategic  Capabilities  Office  (SCO).  The  RCO 
began  in  2003;  its  basic  purpose  is  to  accomplish  “expedited  and  operationally  focused 
concept-through-fielding  activities  to  support  immediate  and  near-term  needs”  (Clark  & 


3  Observation  offered  at  a  symposium  in  May  2015,  not  for  attribution. 
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Freedburg,  2016;  U.S.  Air  Force,  2009).  In  addition,  the  Navy  has  proposed  an  office  which 
will  “be  something  that  closely  mirrors  the  Air  Force  RCO,”  according  to  the  Navy’s  senior 
acquisition  officer  (Clark,  2016). 

The  SCO  originated  in  2012.  Its  basic  purpose  is  “to  re-imagine  existing  DoD  and 
intelligence  community  and  commercial  systems  by  giving  them  new  roles  and  game¬ 
changing  capabilities  to  confound  potential  enemies  (with)  the  emphasis  ...  on  rapidity  of 
fielding”  (Carter,  2016;  Clark  &  Freedburg,  2016).  In  addition,  an  ongoing  legislative 
initiative,  associated  with  Rep.  Mac  Thornberry  (R-TX),  is  intended  to  streamline  acquisition 
processes  (Hasik,  2016b). 

Also,  Secretary  of  the  Air  Force  Deborah  Lee  James  (2015)  has  begun  the  “should 
schedule”  initiative:  “The  previous  incentive  focused  on  cost,  now  we‘d  like  to  target  delivery 
time.  ...  If  we  can  collectively  beat  the  historical  developmental  schedules  and  reward  the 
behavior  in  government  and  industry  that  speeds  things  up,  we  have  a  real  chance  to  make 
a  difference.” 

To  implement  the  “should  schedule,”  Secretary  James  proposes,  inter  alia,  that 
schedule  be  a  major  factor  in  source  selections:  “If  an  industry  partner  can  propose  a 
solution  that  credibly  offers  a  way  to  accelerate  successful  EMD,  then  that  company  would 
have  a  competitive  advantage  for  the  award”  (James,  2015). 

This  sounds  good,  but  let’s  consider  a  future  acquisition  scenario.  Suppose  a  major 
acquisition  program  involves  proposals,  from  Firms  A  and  B,  and  that  Firm  A  wins  the 
competition.  Let’s  further  suppose  that  estimated  schedule  is  a  major  factor  in  that  decision. 
Finally  suppose  this  particular  program  involves  a  long-term,  high-value,  winner-take-all 
contract — like  many  competitions  these  days.  And  it’s  a  safe  bet  that  Firm  B  will  protest.4 
While  “any  accelerated  EMD  plan  would  need  to  survive  a  detailed  scrub  by  independent 
engineers”  (James,  2015),  that  might  well  not  be  enough.  At  minimum,  those  proposed 
schedules  should  also  survive  a  detailed  scrub  by  the  GAO. 

On  Schedule  Estimating  Methods 

Program  schedule  time  can  be  analyzed  and  forecast  according  to  the  following 
(non-inclusive)  menu: 

•  Schedule  length  arising  from  an  orderly  relationship  involving  key  variables; 

•  Schedule  as  a  result  of  a  series  of  management  decisions  intended  to 
produce  the  best  outcome  with  respect  to  performance,  cost  and  time; 

•  Schedule  resulting  from  the  interactions  among  a  set  of  tasks  needed  to 
complete  the  program. 

We  know  quite  a  bit  about  the  last  item — through,  for  example,  Program  Evaluation 
and  Review  Technique,  Critical  Path  Method  and  Gantt  Charts  (Blanchard  &  Fabrycky, 
2006,  esp.  Chap.  1 1 ;  Defense  Systems  Management  College  [DSMC],  2001 ). 


4  The  protest  may  or  may  not  have  a  convincing  rationale.  See,  for  example,  Bill  Sweetman’s  (2015e) 
analysis  of  the  Boeing-Lockheed  Martin  protest  of  the  LRSB  source  selection. 
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We  know  less  about  the  second  but  can  learn  more  through  case  studies  (as 
discussed  below),  and  official  post-mortems  like  those  conducted  for  cost  problems  by  the 
OSD’s  Office  of  Program  Assessment  and  Root  Cause  Analysis  (PARCA)  office.5 

We  know  some  things  about  the  first,  through  descriptive  analyses  such  as  illustrated 
in  Figure  1.  And  we  can  do  more  to  improve  empirical  methods.  Along  those  lines,  the 
discussion  below  provides  an  interesting  empirical  analysis  of  schedule  lengths. 

It’s  possible  to  formulate  an  orderly-relations  approach  in  a  manner  similar  to 
formulating  Cost  Estimating  Relationships.  We  offer  the  term  “Schedule  Estimating 
Relationships.”  We  already  have  a  fair  number  of  possibilities  for  key  explanatory  variables. 
These  include  the  following: 

•  risk  reduction  measures  (including  those  prior  to  source  selection); 

•  contract  type; 

•  technical  maturity  of  subsystems  and  components; 

•  requirements  growth  (or  not); 

•  “complexity”  and  “density”;  and 

•  funding  instability  (or  not). 

Worth  noting  is  that  some  (perhaps  all)  items  on  this  list  could  also  apply  to  program 
cost  estimation. 

There’s  nothing  original  here;  the  first  four  items  have  been  publicly  cited  as  lessons 
learned  and  applied  to  the  LRSB  program  (Butler,  2015;  Seligman  et  al.,  2015;  Sweetman, 
201 5d;  Tirpak,  2015).  In  addition,  below,  we  discuss  requirements  growth  in  the  F-35 
program. 

“Complexity”  is  suggested  as  an  explanatory  variable  by  a  particularly  interesting 
comment  by  a  senior  DoD  official:  “Our  complexity  reach  exceeds  our  engineering  grasp.”6 
One  plausible  metric  for  complexity  is  lines  of  code  (virtual  complexity  perhaps).  For 
example,  Hallion  (1990)  reports  64,000  lines  of  code  in  the  F-15A  and  2.4  million  lines  in  the 
F-15E.  Lines  of  code  in  the  F-35  vary  with  source  and  date.  A  2014  CRS  report  estimates  F- 
35  software  as  containing  approximately  29  million  lines  of  code  and  still  growing  (Gertler, 
2014,  p.  14). 

In  addition  to  virtual  complexity,  we  could  consider  “density,”  indicating  physical 
complexity.  Density  is  “how  tightly  systems  and  equipment  are  placed  within  a  hull  structure” 
(Grant,  2008).  There  is  other  interesting  research  on  “density”  as  a  cost  driver  for  warships 
(e.g.,  Terwilliger,  2015). 


5  In  fact,  post-mortem  analysis  of  schedules  arguably  fits  directly  within  the  PARCA  charter. 
(http://www.acq.osd.mil/parca/performance-assessments.shtml). 

6  Ninth  Annual  Acquisition  Research  Symposium,  May  2012,  Monterey,  CA.  Comment  understood  as 
not  for  attribution. 
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Schedule  Estimating  Goals 

We  think  that  something  like  the  current  structure  for  cost  estimates  is  a  useful 
analog  in  thinking  about  a  similar  structure  for  schedule  estimates.  This  is  illustrated  in 
Figure  2. 


Planning  and 

Preliminary 

Detail  Design 

Production 

Concept  Design 

System  Design 

and 

Development 

Figure  2.  Schedule  Estimation  by  Program  Phase 

(adapted  from  Blanchard  &  Frabrycky,  2006,  Figure  17.9,  p.  595) 

Note.  Source  draft  referred  to  cost  estimation  by  program  phase. 

In  that  vein,  a  comprehensive  schedule  estimating  repertoire  would  include  the 
following: 

•  macro-level,  statistical  methods  to  do  those  estimates  in  the  early  stages  of 
the  program  (ex  ante), 

•  more  specific  methods  to  update  schedule  estimates  during  the  program  (in 
media  res),  and 

•  methods  for  explaining  the  results  of  events  and  decisions  previously  in  the 
program  (ex  post). 

For  estimates  done  early  in  the  program,  we  think  “Schedule  Estimating 
Relationships”  featuring  historically  important  schedule  drivers  are  promising.  They  can 
provide  preliminary  estimates  of  acquisition  schedules  to  inform  concept  and  requirements 
determination.  They  could  also  serve  as  an  independent  check  of  scheduling  aspects  of 
bidders’  proposals. 

During  program  execution,  it’s  highly  desirable  for  program  managers  to  have  the 
means  to  update  schedule  estimates.  To  a  considerable  extent  these  already  exist,  as 
discussed,  for  example,  in  the  DAU’s  Scheduling  Guide  for  Program  Managers  (DSMC, 
2001).  A  number  of  tools  (discussed  above)  are  available  to  program  managers  and  their 
staffs  (Blanchard  &  Fabrycky,  2006,  esp.  Chaps.  11  &  18). 
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Finally,  schedule  analysis  and  prediction  methods  can  usefully  support  after-the-fact 
(ex  post)  analyses  of  program  successes  and  difficulties.  Such  tools  could  enable  schedule 
analyses  similar  to  those  now  conducted  by  the  OSD’s  Root  Cause  Analysis  office  for 
selected  programs  with  cost  problems. 

Cost-Performance-Schedule  Trade-Offs 

Schedules  arise  from  trades  (perhaps  implicit)  among  cost,  schedule  and 
performance.  And  “making  ...  effective  cost,  performance,  schedule,  and  quantity  trade-offs” 
(emphasis  added)  is  a  major  theme  of  the  JCS  directive  for  the  Joint  Capabilities  Integration 
and  Development  System  (JCIDS;  CJCS,  2015). 

We  know  a  fair  amount  about  the  structure  of  cost,  performance,  and  schedule  trade¬ 
offs,  but  there’s  more  worth  finding  out.  As  one  report  put  it,  “the  literature  linking  cost, 
performance,  and  schedule  is  by  no  means  abundant.  This  is  due  in  large  part  to  the  sheer 
complexity  of  the  interrelations  between  performance  characteristics  and  technical 
specifications,  as  well  as  the  unique  missions  ...  systems”  (Voltz,  1992,  p.  13).  In  this 
section,  we  offer  a  preliminary  explanation  of  those  trade-offs  taken  two  at  a  time:  Cost  and 
Performance;  Cost  and  Schedule;  Performance  and  Schedule. 

Cost  and  Performance 

Of  these  three,  we  probably  know  most  about  Cost  vs.  Performance.  Basically,  we 
expect  to  pay  more  to  acquire  higher  performance.  Figure  3  shows  a  notional  trade-off  with 
effects  of  technical  progress. 

That  relationship  has  been  investigated  in  a  number  of  empirical  studies.  One  of 
those  (Hildebrandt  &  Sze,  1986,  p.  15)  led  to  the  following  cost-performance  relationship  (in 
log-log  form)  shown  below.  This  is  the  result  of  a  regression  analysis  of  a  data  base  of  66 
fighter  and  attack  aircraft  with  first  flights  from  1950  (F-89)  to  1979  (F/A-18). 

InCAC  =  1 .99  +  ln*P  +  1 .31  ln*ASP  -  .31  InR  -  .03T  -  ,50*ATTACK  -  ,89*M0D  +  bi*lnY,  (1 ) 

where  CAC  is  cumulative  average  cost;  P  is  resource  price  levels  (primarily  labor 
and  materials);  ASP  is  an  aircraft  performance  index;  R  is  production  rate;  T  is  year  of  first 
flight;  ATTACK  is  dummy  variable  (1  for  attack  aircraft,  0  otherwise);  MOD  is  a  dummy 
variable  for  aircraft  models  that  are  modifications  or  upgrades  of  an  existing  aircraft  type;  bi 
is  the  relevant  learning  curve  parameter;  and  Y  is  cumulative  production.  Franck  (1992) 
used  the  same  data  to  infer  patterns  of  cost-performance  design  choices. 
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Figure  3.  Cost  vs.  Performance  Trade-Off 

(adapted  from  Gansler,  1987,  p.  K-8;  Sullivan,  1981) 

The  first-flight  variable  is  intended  to  capture  the  effects  of  technical  progress.  All 
other  things  equal,  we  expect  to  pay  less  for  a  given  level  of  performance  with 
improvements  in  technology.  This  is  illustrated  in  Figure  3,  which  includes  effects  of 
technical  progress.  As  previously  stated,  as  performance  increases,  so  does  cost.  However 
advancing  technology  shifts  the  cost-performance  curve  down  and  to  the  right  (lessening  to 
cost  of  any  given  level  of  performance). 

Program  Schedule  and  Cost 

Effect  of  development  time  on  program  cost  is  somewhat  ambiguous.  Keeping  a 
team  in  place  longer  means  greater  overhead  expenses  (sometimes  called  the  “standing 
army”  effect).  But  shorter  development  times  can  mean  less  chance  to  develop  technology 
and  sort  among  alternative  approaches  and  incurring  the  costs  associated  with  cascading 
effects  of  wrong  turns. 

These  seem,  in  general,  to  be  countervailing  effects.  Less  time  means  less  overhead 
cost  over  the  life  of  the  program.  More  time  means  better  chances  to  avoid  pitfalls  and 
manage  risk.  In  theory,  the  best  course  of  action  is  reached  by  balancing  increases  in 
overhead  (indirect)  cost  with  direct  program  cost.  This  is  shown  in  Figure  4. 
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Figure  4.  Total  Cost  Analysis  for  Selecting  Optimum  Program  Duration 

(adapted  from  DSMC,  2001,  p.  60;  Zschau,  1969,  pp.  28-30) 

This  sketches  nicely,  but  solving  the  implied  problem  is  more  complicated.  For 
example,  not  all  relevant  costs  are  internal  to  the  acquisition  program  itself.  As  the  DSMC 
Scheduling  Guide  points  out,  “Each  month  added  to  the  development  and  production  of  a 
new  ...  system  tends  to  reduce  by  1  month  the  operational  life  of  the  product”  (DSMC,  2001, 
p.  61).  This  suggests  that  monetized  effects  of  fielding  delay  should  be  added  to  total 
costs — a  difficult  task. 

Nonetheless,  those  costs  of  delay  can  be  all  too  real  and  multifaceted,  as  illustrated 
by  the  F-35  program.  The  effects  included  a  projected  shortfall  of  tactical  fighters  in  both  the 
Air  Force  and  Navy  (Tirpak,  2010;  Trimble,  2010).  To  help  bridge  that  gap,  it  was  necessary 
to  keep  the  older  “legacy”  aircraft  in  service  for  longer  than  originally  planned — and 
consequently  spend  more  money  than  originally  planned  to  retard  their  rate  of 
obsolescence.  For  example,  the  U.S.  Air  Force  has  been  obliged  to  devote  considerable 
resources  to  upgrading  its  “legacy”  fourth-generation  systems  and  to  extending  their 
operational  lives  (GAO,  2012).  Overall,  “the  failure  of  the  so-called  fifth-generation  fighters 
...to  arrive  on  time  and  on  cost  has  cascading  effects  throughout  U.S.  and  allied  fighter 
forces”  (Sweetman,  2012). 

Schedule  and  Performance 

A  notional  representation  of  system  performance  vs.  program  time  appears  in  Figure 
5.  The  figure  implies  that  increasing  program  time  allows  for  a  more  considered  approach 
that  permits  better  decisions.  However,  increases  in  indirect  cost  caused  by  a  longer 
program  crowd  out  resources  directly  useful  for  system  development.  And  beyond  some 
point,  the  slope  of  the  Performance  vs.  Time  curve  goes  negative. 


\\  '  t  )  <■;  ,s,Um  fj 
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Figure  5.  Performance  vs.  Development  Time  Trade-Off 

(adapted  from  Zschau,  1969,  pp.  28-30) 


While  certainly  understandable  in  the  abstract,  there  are  some  difficulties  with  this 
trade-off  in  practice.  Among  other  things,  development  programs  that  proceed  with  a  strictly 
fixed  budget  are  very  rare,  if  not  nonexistent.  This  limits  opportunities  to  develop  a  model 
grounded  in  actual  experience. 

The  Concurrency  Issue 

Program  “concurrency”  is  generally  understood  to  involve  beginning  production  prior 
to  completion  of  development  testing  (DoD,  2015,  p.  46),  or  more  broadly  as  “combining  or 
overlapping  phases”  (“Concurrency,”  n.d.). 

Concurrency  is  frequently  cited  as  a  “high  risk  strategy  that  often  results  in 
performance  shortfalls,  unexpected  cost  increases,  schedule  delays,  and  test  problems” 
(GAO,  2012).  On  the  other  hand,  Goure  (2015)  noted  “a  number  of  reasons  to  pursue 
concurrency,”  including  early  identification  of  production  problems  and  faster  fielding  of  new 
hardware. 

One  very  interesting  study  suggests  an  optimum  level  of  concurrency  (from  the 
perspective  of  cost).  However,  the  authors  did  not  find  strong  empirical  evidence  to  support 
that  hypothesis  (Birchler  et  al.,  201 1 ,  p.  252).  Nonetheless,  some  form  of  dynamic, 
simultaneous-equation  model  might  prove  useful. 

Measuring  Performance 

Metrics  for  cost  and  schedule  time  are  generally  well  understand.  Metrics  for 
performance  are  much  less  definite.  Generally  system  “performance”  is  reported  as  a  vector. 
For  tactical  fighters,  the  elements  of  the  vector  are  characteristics  such  as  maximum  speed, 
service  ceiling,  thrust-to-weight  ratio,  combat  range,  weapons  carriage,  and  Radar  Cross 
Section  (RCS).  One  noteworthy  effort  to  develop  performance  indices  (scalar  measures)  for 
a  variety  of  combat  system  types  was  undertaken  by  the  Analytic  Sciences  Corporation 
(ANSER).  This  occurred  mostly  in  the  1980s  and  described  as  the  TASCFORM  method 
(Regan  &  Voigt,  1988). 

Within  that  overall  project,  the  TASCFORM-Air  model  of  combat  capability  was 
intended  to  assess  tactical  fighters,  attack  helicopters,  and  bombers  in  various  conventional 
missions  (Regan  &  Voigt,  1988,  p.  1-1).  Tactical  aircraft  were  assessed  in  the  context  air-to- 
air  (“air  combat”)  and  “surface  attack”  against  both  land  and  maritime  targets  (p.  2-2).  The 
basic  intent  of  TASCFORM-Air  was  to  systematize  observable  technical  features  and 
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combine  those  with  judgments  of  air  combat  experts  to  provide  (scalar)  indices  of  fighter 
capability  in  several  operational  contexts. 

The  capability  measures  applied  directly  to  individual  aircraft  are  organized  in  a 
hierarchy: 


•  Weapon  Performance  (WP,  a  function  of  weapons  carriage,  range, 
maneuverability  and  speed); 

•  Weapon  System  Performance  (WSP,  WP  plus  target  acquisition, 
susceptibility  to  countermeasures,  weapon  enhancements,  navigation  and 
survivability) 

•  Adjusted  Weapon  System  Performance  (AWSP,  WSP  plus  “obsolescence” 
and  sortie  rate;  p.  2-4). 


So,  how  does  the  F-35  performance  look  relative  to  fourth-generation  fighters?  A 
comparison  of  the  aircraft  types  in  the  Weapons  Performance  dimensions  (which  emphasize 
payload,  range,  maneuverability,  and  speed)  shows  there’s  not  much  difference. 

•  Hard  points:  F-35  has  a  comparable  number  of  weapons  hard  points  relative 
to  the  F-18  and  F-16  but  much  fewer  in  stealth  mode. 

•  Max  Speed:  all  three  aircraft  are  all  comparable. 

•  Ferry  Range  is  comparable,  if  F-35  has  external  tanks. 

•  There’s  a  Combat  Range  advantage  for  the  F-35  when  operating  in  a  high- 
high-high  profile,  compared  with  range  in  a  high-low-high  profile  for  the 
fourth-generation  fighters. 

•  Maneuverability:  Thrust-to-weight  ratio,  max  Gs,  and  wing  loadings  are 
comparable  for  F-16,  F-18,  and  F-35. 

•  Sortie  Rate:  not  yet  determined.  The  F-35  is  still  maturing. 

•  Survivability:  favors  stealthy  aircraft,  but  nonetheless  subject  to 
countermeasures  (discussed  above). 

Force  capability  is  generally  presented  as  numbers  and  types  of  systems.  Force 
capability  indices  are  also  discussed  in  TASCFORM  (pp.  2-4,  2-30-2-36),  in  which  force 
capability  is  assumed  to  be  the  sum  of  individual  performance  (by  tail  number).  These 
measures  do  not  fully  address  force  effectiveness  as  a  function  of  networking  and  shared 
situational  awareness. 

The  fifth-generation  fighter  advocates  have  a  new  perspective  on  system  and  force 
capabilities.  New  aircraft  models  such  as  the  F-22  and  F-35  are  seen  as  disruptive 
innovations.  Within  this  perspective,  the  operational  capabilities  of  the  fifth  generation  are 
due  to  the  combination  (synergy  perhaps)  of  airframe  characteristics  and  “ability  to  work 
within  and  interact  with  a  broad  array  of  networked  systems”  (Deptula,  2011;  Space  Daily 
Staff,  2006). 

Moreover,  fifth-generation  characteristics,  especially  stealth,  increase  the  proportion 
of  resources  devoted  to  offensive  air  operations.  Fifth-generation  aircraft  likely  need  fewer 
fighter  sorties  to  support  penetration  of  advanced  and  integrated  air  defenses,  and  fewer 
tanker  sorties  (due  to  smaller  strike  packages). 

Regardless  of  one’s  opinion  of  fifth-generation  performance  advantages,  it’s  hard  to 
avoid  the  conclusion  that  a  credible  method  of  measuring  system  (and  force)  performance 
should  account  for  the  advantages  of  stealth,  shared  battlefield  awareness,  and  networked 
operations. 
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F-35:  From  CALF  TO  JSF7 


The  Lockheed  Martin  F-35  Joint  Strike  Fighter  (JSF,  Lightning  II)  is  a  single  seat, 
single  engine,  fifth-generation  multirole  fighter  designed  to  perform  ground  attack, 
reconnaisance,  and  air  defense  missions  while  in  stealthy  operation.  It  was  originally 
visualized  as  a  relatively  affordable  strike  fighter  available  in  three  largely  common  versions 
for  the  Air  Force,  Marines,  and  Navy.  It  didn’t  work  out  that  way. 

Then-Major  General  Christopher  Bogdan  (JSF  Program  Executive  Officer  designate) 
commented  that  the  F-35  “is  not  a  single  program  (but  rather)  three  separate  airplane 
programs  (with)  common  avionics  and  a  common  engine.”  He  also  stressed  the  difficulties 
involved  in  reaching  agreement  on  decision-making.  In  his  words,  “lt‘s  hard  enough  to  get 
one  service  to  answer  questions  about  requirements.  Imagine  three  services,  eight  partners, 
and  two  FMS  customers”  (Bogdan,  2012). 

All  three  models  are  designed  for  limited  supersonic  operation,  and  to  carry  their 
primary  weapons  internally,  to  preserve  their  stealth  characteristics.  Although  physical 
differences  arose  from  methods  of  takeoff  and  landing,  requirements  were  also  driven  by 
different  operational  needs. 

The  F-35A  was  a  replacement  for  the  F-16,  the  A-10,  and  perhaps  the  F-15  fighter. 

In  addition,  it  is  intended  to  complement  the  F-22  air  superiority  fighter.  The  Air  Force  sought 
an  advanced  attack  aircraft  with  stealth,  advanced  avionics,  and  low  life-cycle  operating 
costs  providing  improved  range,  speed,  and  appreciable  weapons  load  capacity. 

The  F-35B  is  a  short  takeoff  and  vertical  landing  aircraft  (STOVL)  acquired  to  replace 
its  AV-8B  Harrier  and  its  F/A-18A/B/C/D  strike  fighters.  It  was  designed  to  operate  from 
forward  battlefields,  helicopter  carriers,  and  as  a  “jump  jet”  from  smaller  conventional 
carriers.  The  F-35C  (CV)  chosen  by  the  U.S.  Navy  resembled  the  Air  Force’s  F-35A  but  was 
modified  for  carrier  operations.  It  is  intended  to  replace  earlier  versions  of  the  F/A-18. 

Joint  and  International  Nature 

At  the  time  of  JSF  conception,  there  was  a  clear  preference  at  the  highest  levels  of 
the  DoD  for  joint  projects.  Typically,  the  rationale  for  jointness  is  that  a  largely  joint  project 
lessens  costs  of  developing,  procuring,  and  operating  and  supporting  some  large  number  of 
separate  aircraft  designs  with  similar  (but  not  necessarily  identical)  requirements. 

A  study  by  the  RAND  Corporation  undertook  to  examine  this  issue  (Lorell  et  al., 
2013),  which  focused  on  the  costs  of  jointness.  The  critical  finding  is 

the  need  to  accommodate  different  service  requirements  in  a  single  design  or 
common  design  family  leads  to  greater  program  complexity,  increased 
technical  risk,  and  common  functionality  ...  beyond  that  needed  for  some 
variants,  potentially  leading  to  higher  overall  cost,  despite  the  efficiencies  (of 
common  design).  (Lorell  et  al.,  2013,  iii) 


7  This  section  relies  in  part  on  background  information  from  Aboulafia  (2015)  and  Gertler  (2014).  Also 
we  found  the  Wikipedia  article  on  the  F-35  to  be  a  good  source  for  those  seeking  basic  information  on 
the  program  (“Lockheed  Martin  F-35  Lightning  II,”  n.d.). 
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The  F-35  won  the  DoD  source  selection,  with  an  industry  team  of  Lockheed, 

Northrop  Grumman,  BAE  Systems,  and  Pratt  &  Whitney.  (Aboulafia,  2015,  identified  the  F- 
35  suppliers  in  more  detail.) 

From  the  earliest  days  of  the  JSF  project,  the  Office  of  the  Secretary  of  Defense 
stressed  international  participation.  The  UK  joined  the  JSF  project  as  a  Level  1  Full 
Collaborative  Partner.  There  were  five  (Level  II)  Associate  Partners  and  three  (Level  III) 
Informed  Partners  in  the  Systems  Development  and  Demonstration  Phase  (Schreiber,  2002, 
p.  164). 

History  and  Antecedents 

Defense  procurement  funding  fell  sharply  in  the  early  1990s,  implementing  the 
Bottom  Up  Review  recommendations — ending  such  programs  as  the  NATF  (Naval 
Advanced  Tactical  Fighter)  and  the  A-12/ATA.  Fearing  loss  of  domestic  military  aircraft 
design  skills,  the  DoD  undertook  a  series  of  largely  unsuccessful  programs.  This  effort 
include  support  for  design  of  advanced  technology  aircraft  available  for  production. 

The  list  of  aircraft  concepts  not  leading  to  production  includes  the  following  (e.g., 
Aboulafia,  2015,  esp.  pp.  10-11): 

•  A-X/A/F-X,  a  Navy-dominated  joint  program  was  canceled  due  to  the  A-12’s 
high  cost,  and  by  the  1993  appearance  F/A-18E/F  (Super  Hornet). 

•  ASTOVL/SSF  (Advanced  STOVL/STOVL  Strike  Fighter)  was  an  ARPA 
project  intended  to  develop  a  supersonic  AV-8B  Harrier  successor.  NASA 
and  the  UK  both  participated  in  this  effort.  It  was  merged  by  Congress  with 
JAST  in  mid-1994. 

•  CALF  (Common  Affordable  Lightweight  Fighter)  was  the  formal  name  for 
DARPA’s  ASTOVL  project  that  included  a  conventional  take-off  design 
capability.  Sometimes  known  as  the  X-32,  CALF  was  merged  into  JAST  in 
November  1994. 

•  JAF  (Joint  Attack  Fighter)  explored  the  same  ideas  as  JAST,  as  was  also  true 
of  the  JSSA,  the  Joint  Stealth  Strike  Attack  Aircraft. 

•  MRF  (Multi-Role  Fighter)  was  a  Navy/Air  force  program  designed  to  produce 
a  follow-on  aircraft  for  the  F-16,  F/A-18  and  several  other  legacy  planes.  It 
was  sidetracked  by  the  appearance  of  the  F/A-18E/F  (Super  Hornet). 

JAST/JSF 

The  F-35  Joint  Strike  Fighter  emerged  from  the  Joint  Advanced  Strike  Technology 
Program  (JAST).  However,  JAST’s  original  goal  was  to  develop  technologies  for  advanced 
strike  aircraft  (DSB,  1994).  It  happened  that  JAST’s  plans  to  fund  several  concept 
demonstrator  aircraft  in  1996  coincided  with  ASTOVL’s  planned  timing  of  the  start  of  its 
Phase  III  (full-scale  flight  demonstration).  The  managements  of  both  programs  concluded 
that  it  would  be  logical  to  make  JAST  the  U.S.  military  service  sponsor  for  the  flight 
demonstration  phase  of  ASTOVL.  In  any  case,  FY95  budget  legislation  directed  an 
immediate  merger  of  ASTOVL  into  JAST  (DoD,  JSF  History,  2015). 

In  early  1997,  Lockheed  Martin  and  Boeing  were  selected  to  develop  flying  airframes 
for  the  concept  demonstration  phase.  They  were  designated  X-32  (Boeing)  and  X-35 
(Lockheed  Martin),  respectively,  with  evaluations  between  September  2000  and  August 
2001 .  On  October  26,  2001 ,  the  Lockheed  Martin  team  was  announced  as  the  winner,  after 
which  the  program  transitioned  to  the  JSF  System  Development  and  Demonstration  (SDD) 
phase  (Aboulafia,  2015,  esp.  pp.  11-12). 
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Cost  and  Scheduling  Problems 

Few  new  weapon  systems  have  earned  such  a  widespread  reputation  for  problems 
encountered  in  the  design  and  development  stages  as  the  F-35.  A  few  comments  are  useful 
here.  Early  development  problems  in  many  new  products  were  followed  by  highly  effective 
operational  performance.  The  C-17  is  one  example  (Franck  et  al. ,  2012).  But  the  F-35 
involved  much  more  difficult  design  and  development  problems  (Blickstein  et  al.,  201 1 ,  esp. 
pp.  42,  49). 

The  RAND  Corporation  and  others  reviewed  the  Joint  Strike  Fighter  and  provided  a 
root  cause  analysis  of  its  cost  problems.  The  RAND  report  identified  “in  some  measure”  an 
overly  optimistic  government  estimate  of  the  influence  of  acquisition  reform  and 
“produceability  initiatives”  as  responsible  for  underestimates  of  future  procurement  cost 
growth.  When  combined  with  a  perceived  strong  need  for  an  improved  F-16  replacement, 
the  OSD  proved  willing  to  begin  “a  technologically  complex,  highly  concurrent  F-35 
program.”  The  end  results  included  schedule  slippage  and  cost  growth  that  resulted  in  a 
Nunn-McCurdy  breach  (e.g.,  Rutherford,  2010). 

Explaining  the  Time  Curve 

In  this  section,  empirical  models  are  discussed  that  focus  on  the  key  variable: 

Months  from  Initial  Award  to  IOC  (or  Time  to  IOC)  for  fighter  aircraft.  As  shown  previously, 
this  variable  has  increased  with  later  initial  award  years  for  fighter  aircraft. 

In  this  empirical  analysis,  contract-level  data  contained  in  Selected  Acquisition 
Reports  (SARs)  or  the  F/A-18E/F,  F-22,  and  F-35  (and,  where  possible  the  Air  Force  F-35A) 
is  emphasized.  For  each  published  SAR,  generally  on  December  31 ,  both  program  and 
contract  level  data  are  included.  Because  each  SAR  for  fighter  aircraft  contain  data  for  two 
or  more  contracts  (both  airframe  system  and  engine),  data  at  this  level  not  only  increases 
the  size  of  the  data  set  but  also  permits  inclusion  of  contract  type,  changes  in  the  target 
cost,  years  elapsed  since  time  of  contract  award,  and  contract  variance.  Contract  variance 
information  (not  required  for  Firm  Fixed  Price  contracts)  includes  both  cost  and  schedule 
variance.  The  former  provides  information  on  the  difference  between  the  planned  and  actual 
contract  cost,  the  latter  on  the  difference  between  the  planned  work  performed  and 
scheduled.  (Future  analysis  will  extend  this  work  to  include  program-level  data  including  the 
various  program-level  variances.) 

Examination  of  the  available  data  indicates  that  complex  interactions  among  the 
relevant  variables  complicates  the  traditional  regression  analysis  view  of  explanatory 
variables  affecting  the  dependent  variable.  Our  analysis  includes  both  explanation  and 
association.  Including  association  variables  provides  insight  into  the  strength  of  the 
relationships  between  these  variables  and  the  dependent  variable  (other  variables  held 
constant). 

We  are  also  investigating  professional-judgment  measures  of  fighter  effectiveness 
for  fighters,  which  would  increase  the  regressions’  explanatory  power.  One  variable 
obtained  from  non-SAR  sources,  included  in  the  current  analysis,  is  the  percent  of  an 
aircraft’s  structural  weight  consisting  of  composites. 

To  understand  the  empirical  analysis,  an  influence  diagram  appears  in  Figure  6 — in 
the  form  of  path  analysis  in  which  Time  to  IOC  is  related  to  contract-specific  cost  variance 
and  other  variables.  In  turn,  current  minus  initial  target  cost  is  related  to  contract  variance 
data,  contract  type,  and  several  other  variables.  We  also  identify  the  expected  signs  of  the 
regression  coefficients  when  possible. 
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Broadly  speaking,  Figure  6  displays  those  variables  that  directly  related  to  Months 
from  Initial  Contract  Award  to  IOC,  namely  Contract  Current — Initial  Target  Price,  Program 
Year,  SAR  Year — Contract  Award  Year,  and  Composites  as  a  Percent  of  AC  Structure 
Weight.  There  are  also  indirect  relationships  between  certain  explanatory  variables  and  time 
to  IOC.  This  occurs  through  these  variables’  direct  relationship  with  [Contract  Current — Initial 
Target  Price],  The  variables  with  an  indirect  relationship  with  time  to  IOC  are  Program  Year, 
[SAR  Year — Contract  Award  Year],  [Contract  Cost  Variance  and  Contract  Schedule 
Variance],  Aircraft  MDS,  and  Contract  Type. 

The  only  variables  with  uncertain  sign  of  regression  coefficients  are  Aircraft  MDS  and 
Contract  Type.  It  is  likely  easiest  to  understand  this  diagram  through  a  discussion  of  the 
regression  results.  First,  Figure  6  shows  the  direct  relationship  between  explanatory 
variables  and  Time  to  IOC. 


Figure  6.  Structure  of  the  Regression  Model 

The  results  in  Table  1  show  all  variables  being  statistically  significant  given  the 
hypothesized  signs  of  the  coefficients  in  the  figure.  When  the  current  target  minus  the  initial 
target  price  increases,  this  likely  means  that  a  specification  change  occurred.  One  would 
expect  specification  change  to  be  associated  with  a  longer  Time  to  IOC.  As  Program  Year 
increases,  this  is  likely  related  to  a  longer  program  length.  In  turn,  this  is  likely  related  to  an 
increase  in  Time  to  IOC.  An  increase  in  [SAR  Year — Contract,  Award  Year]  indicates  that  a 
schedule  delay  is  likely. 

The  most  interesting  independent  variable  may  be  [Composites  as  a  Percent  of  AC 
Structural  Weight],  We  show  that  as  this  increases,  which  is  exactly  what  occurred  when 
shifting  from  the  F/A-18E/F  to  the  F-22  to  the  F-35  programs,  the  dependent  variable 
increases.  It  is  known  that  working  with  composite  materials  is  more  complex  than  traditional 
materials,  and  as  result,  can  be  expected  to  increase  the  length  of  the  program  to  IOC. 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-110- 


Table  1.  Direct  Effects  for  Months-to-IOC  Regression 


Explanatory  Variable 

Coefficient 

t-statistic 

(Constant) 

50.409 

12.478 

Contract  Current  -  Initial  Target  Price  ($B) 

1.309 

2.880 

Program  Year 

2.030 

10.015 

SAR  Year  -  ContractAward  Year 

0.986 

3.853 

Composites  as  Percent  of  Structural  Weight 

3.051 

21.242 

FFP  Contract 

12.450 

5.920 

Dependent  Variable:  Months  from  Initial  Contract  Award  to  IOC 

R2  =  0.846;  N  =  164 

We  turn  now  to  the  indirect  effects  regression.  We  have  seen  that  [ Contract ,  Current 
Target — Initial  Target  Price]  has  a  positive  direct  relationship  with  the  key  dependent 
variable.  But  there  are  also  variables  that  have  a  direct  relationship  with  [Contract!  Current 
Target — Initial  Target  Price],  and,  therefore,  an  indirect  relationship  with  [Months  from  Initial 
Contract  Award  to  IOC],  Table  2  displays  this  set  of  regression  results. 

Table  2.  Indirect  Effects  in  the  Regression  Model 


Explanatory  Variable 

Coefficient 

t-statistic 

(Constant) 

3.222 

3.941 

Cumulative  Contract  Cost  Variance 

-4.810 

-4.725 

Cumulative  Contract  Schedule  Variance 

-6.072 

-3.339 

Program  Year 

-0.213 

-3.934 

SAR  Year  -  Contract  Award  Year 

0.168 

3.506 

F-35 

-1.333 

-3.867 

F/A-18E/F 

-1.979 

-4.257 

CPAF  Contract 

-0.860 

-2.019 

Dependent  Variable:  Contract  Current  Target  -  Initial  Target  Price 

R2  =  0.665;  N  =  110;  Financial  Variables,  $B 

The  Cumulative  Contract  Cost  and  Schedule  Variance  coefficients  are  interesting. 
Contract  Cost  Variance  Budgeted  Cost  of  Work  Performed  minus  Actual  Cost  of  Work 
Performed;  and  Schedule  Cost  Variance  equals  Budgeted  Cost  of  Work  Performed  minus 
Budgeted  Cost  of  Work  Scheduled.  So,  as  these  variables  both  increase,  the  extent  to 
which  the  contractor  is  over  budget  and  behind  schedule  decreases.  Therefore,  motivation 
to  revise  target  price  also  decreases,  consistent  with  the  negative  coefficients. 

As  Program  Year  increases,  specifications  become  more  settled,  and  target  price 
revisions  are  less  likely,  consistent  with  the  negative  coefficient.  However,  as  contracts 
become  longer,  requirement  and  specification  changes  become  more  likely. 

One  advantage  of  employing  both  direct  and  indirect  modeling  is  that  one  can  more 
effectively  assess  indirect  effects  of  Aircraft  MDS  and  Contract  Type  on  Time  to  IOC.  Both 
the  F-35  and  the  F/A-18E/F  are  negatively  related  to  [Contract!  Current  Target — Initial  Target 
Price],  For  the  F-35,  this  negative  coefficient  offsets  somewhat  the  positive  relationship 
between  [Contract!  Current  Target — Initial  Target  Price]  and  [Months  from  Initial  Contract 
Award  to  IOC],  Finally,  we  find  that  CPAF  contracts  are  negatively  related  to  [Contract! 
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Current  Target — Initial  Target  Price],  likely  meaning  smaller  increase  in  specifications  that,  in 
turn,  increase  target  price. 


Draft  Research  Agenda 

Acquisition  schedules  are  becoming  more  important.  Therefore,  our  final  aim  in  this 
paper  is  to  propose  a  research  agenda  aimed  at  producing  more  accurate  schedule 
estimates,  particularly  in  major  defense  acquisition  programs. 

Schedule  Estimation  Research:  A  Draft  List  of  Questions  and  Tasks 

1 .  What  is  the  current  state  of  schedule  estimation  and  control?  What’s  needed? 
Where  are  the  gaps? 

o  Interview  subject-matter  experts  regarding  current  state  of  schedule 
analysis,  and  areas  for  improvement. 

2.  How  can  operational  performance  metrics  better  capture  contemporary 
operations? 

o  Update  performance  metrics  for  information-age  warfare.  Start  with 
some  existing  method,  such  as  TASCFORM. 

3.  What  model(s)  best  capture  the  trade-offs  among  program  cost  and 
schedule,  as  well  as  operational  capability  of  fielded  equipment?  Can  those 
models  give  insight  into  “troubled  programs,”  with  difficulties  in  cost, 
schedule,  and  performance? 

o  Analyze  previous  case  studies  (e.g.,  from  Kennedy  School  of 
Government)  for  insights  into  program  schedule  drivers. 

o  Publish  new  case  studies  dealing  with  contemporary  acquisition 
programs,  based,  among  other  things,  on  a  thorough  analysis  of 
relevant  SARs. 

4.  What  estimating  relationships  best  capture  time  to  field  new  hardware?  What 
schedule  drivers  are  generally  most  important? 

o  Based  on  available  data,  formulate  and  empirically  test  models  with 
hypothesized  schedule  drivers. 

o  Formulate  and  test  prediction  markets  for  cost  and  schedule 
problems. 

5.  Is  there  a  prediction  market  design  that  would  produce  useful  information 
about  impending  cost  and  schedule  difficulties? 

o  Design  a  prediction  market  for  defense  acquisition  programs.  Test  it  in 
an  experimental  setting. 

While  this  is  a  very  ambitious  research  program,  it  is  readily  decomposable  into 
smaller  projects.  And  that  we  were  able  to  significantly  advance  the  cost  estimating  state  of 
the  art  suggests  we  can  do  the  same  with  schedule  estimation.  Moreover,  we’d  likely  find 
considerable  insights  from  cost  estimation  methods  useful  for  schedule  estimation. 
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Toward  Realistic  Schedule 

Estimates 


Panel  4 

Chip  Franck,  USAF  Academy 

Greg  Hildebrandt,  USAF  Ret,  Independent 

Consultant 

and  Bud  Udis,  Univ  of  CO,  Boulder 


OUTLINE 


INCREASING  IMPORTANCE  OF  SCHEDULES 

DRAFT  SCHEDULING  ESTIMATING  CLASSIFICATION 

SCHEDULE  ESTIMATING  RELATIONSHIP: 
CANDIDATE  SCHEDULE  DRIVERS 

COST-PERFORMANCE-SCHEDULE  TRADEOFFS 

F-35  CASE  NARRATIVE 

STATISTICAL  "EXPLANATION"  OF  "TIME  CURVE" 

DRAFT  RESEARCH  AGENDA  FOR  MORE  REALISTIC 
SCHEDULE  ESTIMATES 


Schedules  are  now  more  important 

•  MODERNIZATION  WITH  RESTRICTED  BUDGETS 
(2020S) 

•  LONGER  TIME  TO  FIELD  (absolute  &  relative) 

"The  fact  is  that  we  are  slower  than  the  bad  guys." 

•  RAPID  FIELDING  INITIATIVES  (RCO,  SCO,  ...) 

•  SCHEDULE  AS  SIGNIFICANT  SOURCE 
SELECTION  FACTOR  ("SHOULD  SCHEDULE") 


DRAFT  SCHEDULE  ESTIMATING 

CLASSIFICATION 


Orderly  relationship(s)  between  observed 
schedules  and  key  variables  ("Schedule 
Estimating  Relationship") 

Result  of  Management  Decisions  (e.g.,  Cost, 
Schedule,  Performance  Tradeoffs;  alternatively 
case  studies  of  decision  making) 

Analysis  of  a  related  set  of  tasks  (e.g.,  PERT, 
Gantt  Charts) 


SCHEDULE  EST.  RELATIONSHIPS 
Some  Candidate  Explanatory  Variables 

•  Risk  Reduction  Efforts 

•  Contract  Type  (e.g.,  regression  below) 

•  Technical  Maturity  (TRLs  perhaps) 

•  Requirements  Growth  (F-35  narrative 

•  Complexity  (e.g.,  lines  of  code,  "density", 

materials)  "Our  complexity  reach  exceeds  our 

engineering  grasp" 

•  Funding  instability  (or  not) 


TRADEOFFS 


COST  VS  PERFORMANCE  COST  VS.  SCHEDULE 


Sources:  Gansler  (87),  Sullivan  (81)  Sources:  DSMC  (01),  Zschau  (69) 
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F-35:  CASE  NARRATIVE  FOR 
REQUIREMENTS  GROWTH 

•  "ALPHABET  SOUP"  (Aboulafia,  2015)  OF 
STUDIES  AND  CONCEPTS 

•  ...  MERGED  INTO  JAST 

•  ...WHICH  EVOLVED  TO  JSF 

•  WELL-KNOWN  COST  AND  SCHEDULE 
DIFFICULTIES 

" ...  stakeholders  despite  their  best  intentions  can  derail  your 
program."  (Bogdan,  2012) 


"Explaining"  the  Structure  of  Time  Curve  Model 

(F/A-18E/F,  F-22,  F-35  Data) 
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REGRESSION  RESULTS 

$M) 

_  Months  from  Initial 
Contract  Award  to 
IOC  (mos.) 


INDIRECT  MODEL 


(Financial  Variables, 

DIRECT  MODEL 


Explanatory  Variable 

Coefficient 

t- statistic 

(Constant) 

50.409 

12.478 

Contract  Current  Target  -  Initial  Target  Price 

0.001 

2.880 

Program  Year 

2.030 

10.015 

SAR  Year  -  Contract  Award  Year 

0.986 

3.853 

Composites  as  Percent  of  Structural  Weight 

3.051 

21.242 

FFP  Contract 

12.450 

5.920 

Dependent  Variable:  Months  from  Initial  Contract  Award  to  IOC 
R2  =  0.846;  N  =  164 


Contract  Current 
Target  -  Initial 
Target  Price 


Explanatory  Variable 

Coefficient 

t-statistic 

(Constant) 

3.222 

3.941 

Cumulative  Contract  Cost  Variance 

-4.810 

-4.725 

Cumulative  Contract  Schedule  Variance 

-6.072 

-3.339 

Program  Year 

-0.213 

-3.934 

SAR  Year  -  Contract  Award  Year 

0.168 

3.506 

F-35 

-1.333 

-3.867 

F/A-18E/F 

-1.979 

-4.257 

CPAF  Contract 

-0.860 

-2.019 

Dependent  Variable:  Contract  Current  Target  -  Initial  Target  Price 

R2  =  0.665;  N  =  110 
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DRAFT  RESEARCH  AGENDA 

•  ASSESS  CURRENT  STATE  OF  ART  (Comp.  lit. 
review;  SME  interviews).  Identify  methodology 
gaps. 

•  IDENTIFY  FACTORS  MOST  LIKELY  TO  BE  SCHEDULE 
DRIVERS  (empirical  &  case  studies) 

•  REFINE  AND  AMPLIFY  OPERATIONAL 
PERFORMANCE  METRICS  FOR  NETTED  WARFARE 

•  PREDICTION  MARKETS  FOR  PROGRAM 
DIFFICULTIES  (design  and  test) 

•  IMPROVED  MODELS  FOR  COST-PERFORMANCE- 
SCHEDULE  METRICS  AND  TRADE-OFFS  METRICS 

AMBITIOUS  BUT  DECOMPOSABLE 
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